Skip to main content

4  Reproducible Research

Setting up simulation studies that everyone can replicate and reproduce

TipLecture 4

This week’s slides can be found here

4.1 Introduction

Dear all,

This week, it’s all coming together: you will apply your newly acquired markup language skills and build a reproducible research archive from scratch. We will assume that, by now, you’re familiar with Git and GitHub. If that’s not the case, please refer to last weeks’ course materials.

The contents of the research compendium will consist of the simulation study performed by Boulesteix, Groenwold, Abrahamowicz, et al. (2020). In this week’s exercises, you will make the simulation study easier to reproduce. By the end of the exercises you’ve created an improved version of how this simulation may be represented in the online supplement.

All the best,

The teaching team

4.2 Preparation

4.2.1 Required reading

Read Boulesteix, Groenwold, Abrahamowicz, et al. (2020).

Most of the paper may be skimmed through, but make sure to closely read the penultimate section named “An example of a statistical simulation”, since this will be this week’s case study.

4.2.2 Optional reading

The interested reader may also consider reading the original Morris, White, & Crowther (2019) paper, that Boulesteix et al. (2020) is based on. This shouldn’t be too hard, because the target audience of ‘Using simulation studies to evaluate statistical methods’ overlaps significantly with graduate students in statistics. And moreover, you might find it extremely useful for your Master’s thesis!

4.3 Exercise 1

In this exercise you will follow the Utrecht University tutorial on creating reproducible manuscripts: https://mlarpis.github.io/quarto/. Follow the exercise instructions labeled in bold face with Step 0 through Step 6.

In the optional section ‘Rendering’, you may skip most of the instructions. Within this course, there is no need for creating docx documents, using the terminal, and managing multiple file outputs. Still, these might be relevant later in your thesis or future career.

For now, we will only need pdf output, which can be generated with the ‘Render’ button after changing the YAML header:

format:
  pdf: default

Save the research compendium in the dedicated folder in your deliverables folder.

After this exercise, the YAML header for your Quarto document might look something like the code in the footnote1.

4.4 Exercise 2

In this exercise, you will substitute the contents of the research compendium you’ve just created with a replication of Boulesteix et al.’s simulation study. We will start with the same code as the original study, but then make the research fully reproducible with Quarto and renv.

The original simulation script was supplemented as a pdf file. Lucky for you, we’ve already copied all of the R code from their pdf supplement for you. The resulting R script is available from https://github.com/mlarpis/markup/blob/main/documents/week-4/original_script.R.

Likewise, we’ve retrieved the necessary data for you. While the complete database can be accessed through https://wwwn.cdc.gov/nchs/nhanes/continuousnhanes/default.aspx?Begi-nYear=2015, the relevant data files for this exercise can be downloaded directly from https://github.com/mlarpis/markup/blob/main/documents/week-4/raw_data.

Now it’s up to you! Create a reproducible document that contains the simulation study and results. Make sure that this document is part of a fully reproducible research compendium.

  1. Start with the materials from Exercise 1.
  2. Add the data from the original study to the repository.
  3. Read the script from the original study.
  4. Divide the code from the script into clear and concise chunks in the Quarto document. Create helper function script(s) whenever appropriate.
  5. Make the code as reproducible as possible, then run the simulation study.
  6. In Quarto, add some textual description and an interpretation of the simulation study results (e.g. bias, confidence interval coverage).
  7. Add anything else needed to make the research compendium reproducible (e.g. renv).

Hand-in the research compendium by adding it to the dedicated folder in your deliverables repository on GitHub within one week.

Exchange research compendiums with a classmate to make sure that your work is indeed completely reproducible.

  1. ---
title: "My Reproducible Manuscript"
format: 
pdf: default
bibliography: references.bib
---
    ↩︎